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ABSTRACT 

This speech describes some of the experiences 
encountered by an auditor during the audit process and the reactions 
of LEA and project personnel to the audit. The presentation focuses 
on the audit methods as viewed by an educational researcher. The 
discussion also attempts to point out some alternatives to the audit 
process and the problem areas wherein researchers may begin to raise 
questions and provide feedback about the audit concept, (Author) 
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Experience in Educational Auditing - 
An Educator' s Perspective 

Robert W. Consalvo 
Heuristics, Inc. 

Dedham, Massachusetts 

, Paper presented at the 1972 AERA 
Annual Meeting, Chicago, Illinois 



The purpose of my presentation is to describe some of the 
things that have happened and are happening in the Audit Process. 
These perceptions have arisen from our own work in the field and 
from discussions with other auditors. I'd also like to point out 
some of the problems that I see with the Audit Process, and offer 
some suggestions for studying, and perhaps changing it. 

Some experiences first in the pre-audit phase are instructive. 
The pre-audit phase, as described by an earlier speaker, is the 
phase in which the auditor reviews the appropriateness of the eval- 
uation design for the particular project. Some of the activities 
which have led to very positive results during the pre-audit phase 
are touched on in the following exampless 





During a review of an evaluation designed by a professional 
researcher, it was shown that attention focused exclusively on 
the instructional component. There was no process evaluation 
(formative evaluation) built into the design. In this case, the 
auditor was able to offer suggestions for a more complete evalua- 
tion. The changes that followed, especially in providing feed- 
back while the project was going on, substantially increased the 
effectiveness of the evaluation design for that particular project. 

A second kind of problem Involved the lack of specific ob- 
jectives, a deficiency which auditors have often cxted in reviews 
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of evaluation designs. Project directors and staff are often 
encountered who still use only a general type of educational goal 
which evaluators have been trying to get away from for quite a 
while. The auditors have been able to recommend individuals or 
groups to help the project develop performance objectives. Such 
steps have helped get the project on the track, and have assisted 
project evaluators to produce a more meaningful design. 

A third type of problem encountered in pre-audits involved 
not a deficient design, but a design so complex that no one in 
the project could understand it. No one wanted to assist with 
the evaluation, because no one had ever seen anything like it. 

It was much too sophisticated to be useful. In this case, the 
auditor was able to suggest ways to simplify it, so that it be- 

/ 

came more acceptable, and a more useful tool. i 

These have been three kinds of experience in which the pre- 
audit phase has yielded positive results. However, there have V 

been instances of negative effects. In one case I know of, a 
competent evaluator quit— left the job because he felt threatened 
by the audit concept. Now one could speculate, of course, that ^ 

such a reaction was a function of his own personality— that no 
professional should feel threatened by review— but it has hap- 
pened, and I think it results from a negative or carping image 
of the auditor that can get established in that pre-audit phase. 

This adversary image is formed not because of misappre- 
hension that it is the auditor’s sole duty to find things wrong, 
but probably because of the nature of evaluation theory and eval- 
uation plans as they currently are structured. We know that we 
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can take any evaluation design and offer some suggestions or re- 
commendations based on our own philosophy of appropriate evalua- 
tion. We have here the basic issue "What criteria apply to an 
evaluation design?" The issue is similar to the problems which 
we have in rating teachers, or rating the quality of themes, or 
scoring essay tests, — problems which have been prosecuted if not 
persecuted in the measurement field for many years. Further, 
while the auditor can say many good things about a project evalua- 
tion, the odds are that he can say many more bad things about it. 
This circumstance does not necessarily mean that the evaluation 
that has been designed is inadequate. We might instead mean that 
the auditor's criteria for evaluating a project are different, his 
expectations of a design are different, his level of sophistication 
is different, or his perspective is different. Any one of these 
conditions would provide reason enough for an auditor to point 
out discrepancies. Chances are, more than one such factor would 
be operating. 

The attention thus far has been on pre-audit activities, and 
review of the design. The second major component of the audit 
consists of the on-site visits. Generally it appears— at least 
from overt behavior of project personnel— that auditors are well 
received. However, it is soon discovered that everyone is a 
little bit concerned about the auditor's presence. Unobtrusive 
measures have revealed this concern. There is a kind of authority 
invested on you when you come. They wish to document everything 
that they've done, and make sure that the people are available 
for you to talk with, and insure that the classrooms are operating, 
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so that you can get an accurate picture of the project. Often 
the fashion in which the evaluator has his data spread out, ready 
for you to inspect, is enough to make it pretty clear that the 
adversary image is operative. 

The most serious problem in the on-site portion of the audit 
Is the usually limited amount of time available (especially on 
smaller projects.) Many bilingual projects are small projects— 
approximately $100,000. Given real fiscal constraints, the limited 
amount of time which can be spent on-site severly restricts the 
kinds and amount of data an auditor can collect. In a fiscal 
audit such restrictions generally do not apply, because the audi- 
tor is empowered to do what he must to certify the books. A fis- 
cal audit can be fairly expensive, if you want to compare it to / 

the general accounting cost for a firm. 

In an educational audit, the decision on what and when to 
sample, or what kind and how much data to collect is very serious, n 
since error introduced at this point will be reflected in the con- 
clusions that the auditor is able to draw. It is an area of 
serious concern, and one which I feel is perhaps the weakest point 
in the audit concept as it stands now. I do not think it is feas- 

’ t 

ible, in most cases, for an auditor to sample well enough to find 
out whether the evaluation has in fact been successful. Of course, 
he could sample actual test scores from the data bank of the pro- 
ject. But if he really is to verify results, he has to collect 
his own data in order to come up with some kind of Independent 
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estimate of the validity of the evalution. That is what takes 
time, manpower, and money. 

The third major area of the audit is the post-audit phase. 

During this phase, the auditor reviews the final evaluation re- 
port, and checks for discrepancies between his own data and the 
data presented by the evaluator. He also looks into the appro- 
priateness of the presentation and the correspondence between 
conclusions and data. Most of our papers, reports, and presen- 
tations should similarly be subject to peer review of some sort. 

It is easy to argue beyond the data, and to present the findings 
in a format which ignores readers less competent in technical 
matters. On some of the projects in which we have been audited, 
the auditor has suggested some interesting ways in which we might 
have otherwise analyzed the data, and ways in which we might 
have presented it a little more clearly. It is interesting, how- 
ever, that in these instances the auditor had not suggested these 
kind of things during the pre-audit. Here is where additional 
difficulty may arise, in reviewing a final report it can become 
evident that other things could have and should have been done 
in the evaluation, but they are things which do not come out until 
one has the data in hand and is working with it. Should the audi- 
tor then comment on a type of analysis or presentation different 
from what he approved in the pre-audit? 

The most serious problem in the post-audit phase is again 
tied in with the problem of collecting data for verification. The 
kind of conclusions the auditor can draw are directly related to 
his sampling procedures* Suppose he has sampled ten students for 
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retesting , out of 30 in a program. Could he draw a conclusion 
opposite to that in the evaluation when the extent of his testing 
does not correspond to the evaluator's? What X am getting at is 
that the sampling error introduced within the on-site phase makes 
it very difficult, I believe, for the auditor to draw honest con- 
clusions regarding the verification of the data analysis. He can 
take existing data and simply do the statistical tests over again, 
but this is hardly the same as providing evidence that the results 
of two independent investigators are the same. If the auditor 
does this reanalysis for one objective, but not another, can he 
justify the assumption that the other objective has been validly 
evaluated — especially when, in the pre-audit stage, during the 
development of the audit plan, the auditor must specify the ob- 
jectives he wishes to audit? 

Another real problem which has arisen in discussions of the 
audit reports with the directors of projects and evaluators is the 
tone of the auditor's final report. Many auditors I know have 
honestly attempted to balance the positive and negative aspects. 
However, the document is generally perceived as being by nature 
a negative document,, There is apprehension before its arrival. 
People wait at the mail for the final audit report to see what 
the auditor has pointed out as deficiencies, because it seems 
pretty clear that the purpose is to point out deficiencies. The 
guidelines make that fairly clear. Project personnel at the time 
of the post-audit report are most anxious about the effects of the 
audit upon their project. The usual reaction by project personnel 
is to call the auditor a Monday morning quarterback. Of course 
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we all know the kinds of problems that arise when one is trying 
to carry out an evaluation design. It's the real world of evalua- 
tion versus the theoretical world of evaluation. If the auditor 
approaches the final audit from a theoretical viewpoint, it is 
quite obvious that in many cases he will be able to do a devas- 
tating job criticizing the evaluation. 

It's quite obvious, I'm sure, that I view the audit concept 
as having problems in each major phase. I think the pre-audit 
is a very important function, and that no evaluator should de- 
sign an evaluation without obtaining an expert review of his 
design. 

* 

I suspect that the on-site work is that phase most likely to 
introduce error into the audit process. I feel that the post- 
audit phase can be valuable to the extent, that it determines 
whether the evaluation is understandable, and whether the con- 
elusions are drawn accurately from the data--just as anyone would 
generally criticize a piece of research, or a paper in a pro- 
fessional journal, 

To turn now to some of the more general problems and promise 
that I see within the audit concept as the state of the art exists 
today. First, the competencies that are required of auditors, as 
has been pointed out by other speakers do not exist in single 
individuals, therefore, the team approach is necessary. Once we 
introduce the team approach, I think we raise the issue of in- 
creasing the cost oi the audit. Hiring several specialized person- 
nel to perform this kind of work and to get them to work as ateam 
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costs a little more money. Where do we get this money? Must we 
divert it from another important aspect of the program or the 
evaluation process? Perhaps if we took the audit money and put it 
into more comprehensive evaluation, it might perform exactly the 
same function — that of increasing the validity of the evaluations 
that are produced. 

Another issue raised by the very concept of adding another 
cadre of qualified personnel to education is — where do we get 
auditors? We all know there is supposed to be a shortage of good 
evaluators. It was one of the intentions of Title IV training 
grants, and it is also the intent of the audit concept, to insure 
that the inadequate evaluations of past Title I and III projects 
are not encountered again. Adding this fund of theoretical talent 
must somehow be reconciled with the fact that we do not now have 
enough people to do the actual evaluations . 

A further quite realistic problem is — who determines which 
auditors are qualified tc audit which evaluators? In the fiscal 
auditing field it is quite clear that Certified Public Accountants 
have the authority to perform audits, and under the regulations 
of their society they can, in fact, certify such processes. There- 
fore, they are at least equal to, and perhaps a professional notch 
above# the people they are auditing. In the case of social science 
auditing or educational auditing, how do we determine whether one 
individual is qualified to audit the work of another? 

There is also the question whether the theory of evaluation 
is so well developed that there is a consensus of what constitutes 
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a good evaluation. Again I note the influence of the negative 
views of the auditor that are so often held by project personnel. 

I think these views limit seriously the impact that the auditor 
can have upon a project. It seems obvious th^t people who are 
viewed negatively are not considered great facilitators of change, 
and if the audit is supposed to bring about change in the project, 
it is important that the present image be changed. 

A serious issue is the reliability of the audit. This is a 
function chiefly of the kinds of data that can be collected in 
the on-site phase, but it also relates to the reliability of the 
other phases. Is the total picture provided by the audit an ac- 
curate picture of the evaluation? We have never really checked 
on this, and I think that we have to. 

Finally, the most important issue, or at least the most prac- 
tical issue, concerns the use to which the audit information is 
put by all levels of the education hierarchy, from the project 
director and the evaluator to the Office of Education and Congress. 
The purpose of the audit is to provide information to insure a 
valid evalution, and we need evidence that the work of the audi- 
tors, is in fact, promoting such evaluation. 

I would like in closing to propose several alternatives to the 
audit process as it is now constituted. First I would reduce the 
scope of the audit, and draw it away from the model of the fiscal 
auditors. I propose that it move more toward a peer/professional 
review of the evaluation design and the final evaluation product. 
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Concurrently, X would increase the duties of the staff members of 
Office of Education in performing these functions. That is, they 
could either perform them completely in house, or in conjunction 
with a team of specialists. The final evaluation design then would 
be approved by all concerned. Scarvia Anderson of ETS has said 
that "the demand for independent audit seems to be directly related 
to the distance between the program and the funding source" (1971, 
p. K-7) . Perhaps we should then just turn it around and bring the 
projects closer to Washington, or Washington closer to the pro- 
jects, so that the need for the audit can be reduced. I also 
think that using only OE personnel adds an air of authority to 
the audit, and tends to eliminate one more level between the 
funding source and the operation of the project. 



Another approach--and one which again may eliminate the use of 
auditors— is simply to place more rigorous requirements on the 
qualifications of evaluators. If somehow we can establish some 
kind of criteria for what kinds of people should be evaluating 
projects, perhaps we could reduce the need for a formal audit 
process as it is now specified. Another alternative would be to 
establish professional teams analogous to proposal review teams 
in the regional HEW offices, and, using some kind of consensus, 
determine the appropriateness of the evaluation and whether it 
has, in fact, been carried out accurately. 

Finally, there is an alternative which X think is most im- 
portant if the Office of Education is going to retain the audit 
concept basically as it is now, and that is to assign auditors to 
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projects at random from the Offic of Education, with changes in 
auditors periodically (yearly or every two or three years) . A 
problem not previously noted is the difficulty the auditor has 
in remaining independent, if he must maintain his level of ap- 
proval with the project high enough so that he is re-hired. The 
auditor would probably tend to be more objective if he were working 
with the U.S. Office of Education, and assigned to audit a parti- 
cular project with an OE staff member, rather than being hired by 
the project to find discrepancies in that same project. 

I have tried to point up some of the good things that I see 
going on in the audit process, some of the problems, and some 
possible alternatives to the audit concept. I recognize that 
the audit concept is new, and therefore in a state of develop- 
ment. Very few good concepts of such scope, I think, spring 
full grown upon the scene. We should recognize that we have 
here an emerging concept, one that may in fact bring about real 
changes in education. We should give it more time to develop, 
but we should not hesitate meanwhile to subject it to the same 
careful and intensive study that we expect of the auditor him- 
self* 
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